Face Generation¶
In this project, you'll define and train a Generative Adverserial network of your own creation on a dataset of faces. Your goal is to get a generator network to generate new images of faces that look as realistic as possible!
The project will be broken down into a series of tasks from defining new architectures training adversarial networks. At the end of the notebook, you'll be able to visualize the results of your trained Generator to see how it performs; your generated samples should look like fairly realistic faces with small amounts of noise.
Get the Data¶
You'll be using the CelebFaces Attributes Dataset (CelebA) to train your adversarial networks.
This dataset has higher resolution images than datasets you have previously worked with (like MNIST or SVHN) you've been working with, and so, you should prepare to define deeper networks and train them for a longer time to get good results. It is suggested that you utilize a GPU for training.
Pre-processed Data¶
Since the project's main focus is on building the GANs, we've done some of the pre-processing for you. Each of the CelebA images has been cropped to remove parts of the image that don't include a face, then resized down to 64x64x3 NumPy images. Some sample data is show below.
If you are working locally, you can download this data by clicking here
This is a zip file that you'll need to extract in the home directory of this notebook for further loading and processing. After extracting the data, you should be left with a directory of data processed-celeba-small/.
# run this once to unzip the file
##!unzip processed-celeba-small.zip
from glob import glob
from typing import Tuple, Callable, Dict
import matplotlib.pyplot as plt
import numpy as np
import torch
from PIL import Image
from torch.utils.data import DataLoader, Dataset
from torchvision.transforms import Compose, ToTensor, Resize, Normalize
import tests
data_dir = 'processed_celeba_small/celeba/'
Data pipeline¶
The CelebA dataset contains over 200,000 celebrity images with annotations. Since you're going to be generating faces, you won't need the annotations, you'll only need the images. Note that these are color images with 3 color channels (RGB)#RGB_Images) each.
Pre-process and Load the Data¶
Since the project's main focus is on building the GANs, we've done some of the pre-processing for you. Each of the CelebA images has been cropped to remove parts of the image that don't include a face, then resized down to 64x64x3 NumPy images. This pre-processed dataset is a smaller subset of the very large CelebA dataset and contains roughly 30,000 images.
Your first task consists in building the dataloader. To do so, you need to do the following:
- implement the get_transforms function
- create a custom Dataset class that reads the CelebA data
Exercise: implement the get_transforms function¶
The get_transforms function should output a torchvision.transforms.Compose of different transformations. You have two constraints:
- the function takes a tuple of size as input and should resize the images to the input size
- the output images should have values ranging from -1 to 1
def get_transforms(size: Tuple[int, int]) -> Callable:
""" Transforms to apply to the image."""
# TODO: edit this function by appening transforms to the below list
transforms = [Resize(size),
ToTensor(),
Normalize(mean= (0.5, 0.5, 0.5), std= (0.5, 0.5, 0.5))]
return Compose(transforms)
Exercise: implement the DatasetDirectory class¶
The DatasetDirectory class is a torch Dataset that reads from the above data directory. The __getitem__ method should output a transformed tensor and the __len__ method should output the number of files in our dataset. You can look at this custom dataset for ideas.
class DatasetDirectory(Dataset):
"""
A custom dataset class that loads images from folder.
args:
- directory: location of the images
- transform: transform function to apply to the images
- extension: file format
"""
def __init__(self,
directory: str,
transforms: Callable = None,
extension: str = '.jpg'):
# TODO: implement the init method
self.directory = directory
self.transforms = transforms
self.extension = extension
self.data = []
for img in glob('{}/*{}'.format(self.directory, self.extension)):
self.data.append(Image.open(img))
def __len__(self) -> int:
""" returns the number of items in the dataset """
# TODO: return the number of elements in the dataset
return self.data.__len__()
def __getitem__(self, index: int) -> torch.Tensor:
""" load an image and apply transformation """
# TODO: return the index-element of the dataset
return self.transforms( self.data[index] )
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
# run this cell to verify your dataset implementation
dataset = DatasetDirectory(data_dir, get_transforms((64, 64)))
tests.check_dataset_outputs(dataset)
The functions below will help you visualize images from the dataset.
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
def denormalize(images):
"""Transform images from [-1.0, 1.0] to [0, 255] and cast them to uint8."""
return ((images + 1.) / 2. * 255).astype(np.uint8)
# plot the images in the batch, along with the corresponding labels
fig = plt.figure(figsize=(20, 4))
plot_size=20
for idx in np.arange(plot_size):
ax = fig.add_subplot(2, int(plot_size/2), idx+1, xticks=[], yticks=[])
img = dataset[idx].numpy()
img = np.transpose(img, (1, 2, 0))
img = denormalize(img)
ax.imshow(img)
Model implementation¶
As you know, a GAN is comprised of two adversarial networks, a discriminator and a generator. Now that we have a working data pipeline, we need to implement the discriminator and the generator.
Feel free to implement any additional class or function.
Exercise: Create the discriminator¶
The discriminator's job is to score real and fake images. You have two constraints here:
- the discriminator takes as input a batch of 64x64x3 images
- the output should be a single value (=score)
Feel free to get inspiration from the different architectures we talked about in the course, such as DCGAN, WGAN-GP or DRAGAN.
Some tips¶
- To scale down from the input image, you can either use
Conv2dlayers with the correct hyperparameters or Pooling layers. - If you plan on using gradient penalty, do not use Batch Normalization layers in the discriminator.
import torch.nn as nn
from torch.nn import Module
import torch.nn.functional as F
class Discriminator(Module):
def __init__(self):
super(Discriminator, self).__init__()
self.conv1 = nn.Conv2d(in_channels=3,
out_channels=128,
kernel_size=5,
stride=2,
padding=2) # Size 32x32
self.conv2 = nn.Conv2d(in_channels=128,
out_channels=256,
kernel_size=5,
stride=2,
padding=2) # Size 16x16
self.batchnorm2 = nn.BatchNorm2d(256)
self.conv3 = nn.Conv2d(in_channels=256,
out_channels=512,
kernel_size=5,
stride=2,
padding=2) # Size is 8x8
self.batchnorm3 = nn.BatchNorm2d(512)
self.conv4 = nn.Conv2d(in_channels=512,
out_channels=1024,
kernel_size=5,
stride=2,
padding=2) # Size is 4x4
self.batchnorm4 = nn.BatchNorm2d(1024)
self.fc1 = nn.Linear(in_features=4*4*1024,
out_features=1)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# TODO: implement the forward method
x = F.leaky_relu(self.conv1(x), 0.2)
x = F.leaky_relu(self.batchnorm2(self.conv2(x)), 0.2)
x = F.leaky_relu(self.batchnorm3(self.conv3(x)), 0.2)
x = F.leaky_relu(self.batchnorm4(self.conv4(x)), 0.2)
x = torch.flatten(x, 1)
x = self.fc1(x)
x = torch.unsqueeze(x, 2) # 2d to 4D conversion
x = torch.unsqueeze(x, 3) # 2d to 4D conversion
return x
# class Discriminator(nn.Module):
# def __init__(self, conv_dim: int = 64):
# super(Discriminator, self).__init__()
# self.main = nn.Sequential(
# # input is ``(nc) x 64 x 64``
# nn.Conv2d(3, conv_dim, 4, 2, 1, bias=False),
# nn.LeakyReLU(0.2, inplace=True),
# # state size. ``(ndf) x 32 x 32``
# nn.Conv2d(conv_dim, conv_dim * 2, 4, 2, 1, bias=False),
# nn.BatchNorm2d(conv_dim * 2),
# nn.LeakyReLU(0.2, inplace=True),
# # state size. ``(ndf*2) x 16 x 16``
# nn.Conv2d(conv_dim * 2, conv_dim * 4, 4, 2, 1, bias=False),
# nn.BatchNorm2d(conv_dim * 4),
# nn.LeakyReLU(0.2, inplace=True),
# # state size. ``(ndf*4) x 8 x 8``
# nn.Conv2d(conv_dim * 4, conv_dim * 8, 4, 2, 1, bias=False),
# nn.BatchNorm2d(conv_dim * 8),
# nn.LeakyReLU(0.2, inplace=True),
# # state size. ``(ndf*8) x 4 x 4``
# nn.Conv2d(conv_dim * 8, 1, 4, 1, 0, bias=False),
# nn.Sigmoid()
# )
# def forward(self, input):
# return self.main(input)
discriminator = Discriminator()
print(discriminator)
images = torch.randn(1, 3, 64, 64)
score = discriminator(images)
print("Shape out = ", score.shape)
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
# run this cell to check your discriminator implementation
discriminator = Discriminator()
tests.check_discriminator(discriminator)
Exercise: create the generator¶
The generator's job creates the "fake images" and learns the dataset distribution. You have three constraints here:
- the generator takes as input a vector of dimension
[batch_size, latent_dimension, 1, 1] - the generator must outputs 64x64x3 images
Feel free to get inspiration from the different architectures we talked about in the course, such as DCGAN, WGAN-GP or DRAGAN.
Some tips:¶
- to scale up from the latent vector input, you can use
ConvTranspose2dlayers - as often with Gan, Batch Normalization helps with training
class Generator(Module):
def __init__(self, latent_dim: int):
super(Generator, self).__init__()
# TODO: instantiate the different layers
self.first_conv_in_channels = 1024
first_conv_out_channels = 512
second_conv_out_channels = 256
third_conv_out_channels = 128 # rgb
fourth_conv_out_channels = 3 # rgb
self.fc1 = nn.Linear(in_features=latent_dim,
out_features=4*4*1024)
self.transpose_conv1 = nn.ConvTranspose2d(
in_channels=1024,
out_channels=512,
kernel_size=2,
padding=0,
stride=2)
self.batch_norm1 = nn.BatchNorm2d(512)
self.transpose_conv2 = nn.ConvTranspose2d(
in_channels=512,
out_channels=256,
kernel_size=2,
padding=0,
stride=2)
self.batch_norm2 = nn.BatchNorm2d(256)
self.transpose_conv3 = nn.ConvTranspose2d(
in_channels=256,
out_channels=128,
kernel_size=2,
padding=0,
stride=2)
self.batch_norm3 = nn.BatchNorm2d(128)
self.transpose_conv4 = nn.ConvTranspose2d(
in_channels=128,
out_channels=3,
kernel_size=2,
padding=0,
stride=2)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# TODO: implement the forward method
x = torch.flatten(x, 1)
x = self.fc1(x)
batch_size = len(x)
x = x.view(batch_size, 1024, 4, 4) # order based on https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html?highlight=convtranspose2d#torch.nn.ConvTranspose2d
x = F.relu(self.batch_norm1(self.transpose_conv1(x)))
x = F.relu(self.batch_norm2(self.transpose_conv2(x)))
x = F.relu(self.batch_norm3(self.transpose_conv3(x)))
x = F.tanh(self.transpose_conv4(x))
return x
# class Generator(nn.Module):
# def __init__(self, latent_dim: int, conv_dim: int = 64):
# super(Generator, self).__init__()
# self.main = nn.Sequential(
# # input is Z, going into a convolution
# nn.ConvTranspose2d( latent_dim, conv_dim * 8, 4, 1, 0, bias=False),
# nn.BatchNorm2d(conv_dim * 8),
# nn.ReLU(True),
# # state size. ``(ngf*8) x 4 x 4``
# nn.ConvTranspose2d(conv_dim * 8, conv_dim * 4, 4, 2, 1, bias=False),
# nn.BatchNorm2d(conv_dim * 4),
# nn.ReLU(True),
# # state size. ``(ngf*4) x 8 x 8``
# nn.ConvTranspose2d( conv_dim * 4, conv_dim * 2, 4, 2, 1, bias=False),
# nn.BatchNorm2d(conv_dim * 2),
# nn.ReLU(True),
# # state size. ``(ngf*2) x 16 x 16``
# nn.ConvTranspose2d( conv_dim * 2, conv_dim, 4, 2, 1, bias=False),
# nn.BatchNorm2d(conv_dim),
# nn.ReLU(True),
# # state size. ``(ngf) x 32 x 32``
# nn.ConvTranspose2d( conv_dim, 3, 4, 2, 1, bias=False),
# nn.Tanh()
# # state size. ``(nc) x 64 x 64``
# )
# def forward(self, input):
# return self.main(input)
latent_dim = 128
lv = torch.randn(1, latent_dim, 1, 1)
generator = Generator(latent_dim)
print(generator)
image = generator(lv)
print("Shape out = ", image.shape)
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
# run this cell to verify your generator implementation
latent_dim = 128
generator = Generator(latent_dim)
tests.check_generator(generator, latent_dim)
Optimizer¶
In the following section, we create the optimizers for the generator and discriminator. You may want to experiment with different optimizers, learning rates and other hyperparameters as they tend to impact the output quality.
Exercise: implement the optimizers¶
import torch.optim as optim
def create_optimizers(generator: Module, discriminator: Module):
""" This function should return the optimizers of the generator and the discriminator """
# TODO: implement the generator and discriminator optimizers
g_optimizer = optim.Adam(generator.parameters(),lr = 0.0002,betas = (0.5,0.999))
d_optimizer = optim.Adam(discriminator.parameters(),lr = 0.0002,betas = (0.5,0.999))
return g_optimizer, d_optimizer
Losses implementation¶
In this section, we are going to implement the loss function for the generator and the discriminator. You can and should experiment with different loss function.
Some tips:
- You can choose the commonly used the binary cross entropy loss or select other losses we have discovered in the course, such as the Wasserstein distance.
- You may want to implement a gradient penalty function as discussed in the course. It is not required and the code will work whether you implement it or not.
Exercise: implement the generator loss¶
The generator's goal is to get the discriminator to think its generated images (= "fake" images) are real.
def generator_loss(fake_logits):
""" Generator loss, takes the fake scores as inputs. """
# TODO: implement the generator loss
# Label Smoothing
labels = torch.ones(fake_logits.size())*0.9
labels = labels.cuda()
criterion = torch.nn.BCEWithLogitsLoss()
loss = criterion(fake_logits.squeeze(), labels.squeeze())
return loss
Exercise: implement the discriminator loss¶
We want the discriminator to give high scores to real images and low scores to fake ones and the discriminator loss should reflect that.
def discriminator_loss(real_logits, fake_logits):
""" Discriminator loss, takes the fake and real logits as inputs. """
# TODO: implement the discriminator loss
criterion = torch.nn.BCEWithLogitsLoss()
# real_loss
labels = torch.ones(real_logits.size())*0.9
labels = labels.cuda()
real_loss = criterion(real_logits.squeeze(), labels.squeeze())
# fake_loss
labels = torch.zeros(fake_logits.size())
labels = labels.cuda()
fake_loss = criterion(fake_logits.squeeze(), labels.squeeze())
loss = real_loss + fake_loss
return loss
Exercise (Optional): Implement the gradient Penalty¶
In the course, we discussed the importance of gradient penalty in training certain types of Gans. Implementing this function is not required and depends on some of the design decision you made (discriminator architecture, loss functions).
def gradient_penalty(discriminator, real_samples, fake_samples):
""" This function enforces """
gp = 0
# TODO (Optional): implement the gradient penalty
return gp
Training¶
Training will involve alternating between training the discriminator and the generator. You'll use your functions real_loss and fake_loss to help you calculate the discriminator losses.
- You should train the discriminator by alternating on real and fake images
- Then the generator, which tries to trick the discriminator and should have an opposing loss function
Exercise: implement the generator step and the discriminator step functions¶
Each function should do the following:
- calculate the loss
- backpropagate the gradient
- perform one optimizer step
def generator_step(batch_size: int, latent_dim: int,
generator: Module,
discriminator: Module,
g_optimizer: optim.Adam) -> Dict:
""" One training step of the generator. """
# TODO: implement the generator step (foward pass, loss calculation and backward pass)
# Generate fake images
fakes = np.random.uniform(-1, 1, size=(batch_size, latent_dim))
fakes = torch.from_numpy(fakes).float()
fakes = fakes.cuda()
fake_images = generator(fakes)
fake_images = fake_images.cuda()
g_optimizer.zero_grad()
# perform backprop
g_loss = generator_loss(discriminator(fake_images))
g_loss.backward()
g_optimizer.step()
return {'loss': g_loss}
def discriminator_step(batch_size: int, latent_dim: int, real_images: torch.Tensor,
generator: Module,
discriminator: Module,
d_optimizer: optim.Adam) -> Dict:
""" One training step of the discriminator. """
real_images = real_images.cuda()
d_optimizer.zero_grad()
# Generate fake images
fakes = np.random.uniform(-1, 1, size=(batch_size, latent_dim))
fakes = torch.from_numpy(fakes).float()
fakes = fakes.cuda()
fake_images = generator(fakes)
fake_images = fake_images.cuda()
d_loss = discriminator_loss(real_logits=discriminator(real_images),
fake_logits=discriminator(fake_images))
d_loss.backward()
d_optimizer.step()
gp = 0 # did not implement gradient penalty
return {'loss': d_loss, 'gp': gp}
Main training loop¶
You don't have to implement anything here but you can experiment with different hyperparameters.
from datetime import datetime
# you can experiment with different dimensions of latent spaces
latent_dim = 128
# update to cpu if you do not have access to a gpu
device = 'cuda'
# number of epochs to train your model
n_epochs = 50
# number of images in each batch
batch_size = 64
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
print_every = 50
# Create optimizers for the discriminator D and generator G
generator = Generator(latent_dim).to(device)
discriminator = Discriminator().to(device)
g_optimizer, d_optimizer = create_optimizers(generator, discriminator)
dataloader = DataLoader(dataset,
batch_size=64,
shuffle=True,
num_workers=4,
drop_last=True,
pin_memory=False)
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
def display(fixed_latent_vector: torch.Tensor):
""" helper function to display images during training """
fig = plt.figure(figsize=(14, 4))
plot_size = 16
for idx in np.arange(plot_size):
ax = fig.add_subplot(2, int(plot_size/2), idx+1, xticks=[], yticks=[])
img = fixed_latent_vector[idx, ...].detach().cpu().numpy()
img = np.transpose(img, (1, 2, 0))
img = denormalize(img)
ax.imshow(img)
plt.show()
Exercise: implement the training strategy¶
You should experiment with different training strategies. For example:
- train the generator more often than the discriminator.
- added noise to the input image
- use label smoothing
Implement with your training strategy below.
fixed_latent_vector = torch.randn(batch_size, latent_dim, 1, 1).float().cuda()
losses = []
d = 0
g = 0
for epoch in range(n_epochs):
for batch_i, real_images in enumerate(dataloader):
real_images = real_images.to(device)
####################################
# TODO: implement the training strategy
####################################
g_loss = generator_step(batch_size, latent_dim, generator, discriminator, g_optimizer)
d_loss = discriminator_step(batch_size, latent_dim, real_images,
generator, discriminator, d_optimizer)
if batch_i % print_every == 0:
# append discriminator loss and generator loss
d = d_loss['loss'].item()
g = g_loss['loss'].item()
losses.append((d, g))
# print discriminator and generator loss
time = str(datetime.now()).split('.')[0]
print(f'{time} | Epoch [{epoch+1}/{n_epochs}] | Batch {batch_i}/{len(dataloader)} | d_loss: {d:.4f} | g_loss: {g:.4f}')
# display images during training
generator.eval()
generated_images = generator(fixed_latent_vector)
display(generated_images)
generator.train()
Training losses¶
Plot the training losses for the generator and discriminator.
"""
DO NOT MODIFY ANYTHING IN THIS CELL
"""
fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator', alpha=0.5)
plt.plot(losses.T[1], label='Generator', alpha=0.5)
plt.title("Training Losses")
plt.legend()
Question: What do you notice about your generated samples and how might you improve this model?¶
When you answer this question, consider the following factors:
- The dataset is biased; it is made of "celebrity" faces that are mostly white
- Model size; larger models have the opportunity to learn more features in a data feature space
- Optimization strategy; optimizers and number of epochs affect your final result
- Loss functions
Answer:
- Low resolution in ==> unable to extract all features.
- DCGAN with BCEWithLogitsLoss worked to produce images that were decent within 50 epochs of training.
- Faces with glasses were difficult to replicate ==> need more training data.
- Blurred eyes were observed ==> lower resolution could be the reason.
- Blurred areas where hair and skin transitions were seen ==> Feature extraction was not great ==> higher resolution could've helped
- Gradient penalty could've helped if using BCE loss
- Hyperparameters: did not optimize past what was already provided
Submitting This Project¶
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_face_generation.ipynb".
Submit the notebook using the SUBMIT button in the bottom right corner of the Project Workspace.